Skip to content

Move ARM64 Helix container hosts to Azure Linux 4 - #136387

Merged
steveisok merged 2 commits into
dotnet:mainfrom
steveisok:steveisok-arm64-helix-container-hosts
Oct 24, 2026
Merged

steveisok merged 2 commits into
dotnet:mainfrom
steveisok:steveisok-arm64-helix-container-hosts

Conversation

@steveisok

Copy link
Copy Markdown
Member

Summary

Native ARM64 container-host equivalent of #134819: move the physical Helix host queues from Azure Linux 3 to Azure Linux 4 without moving standalone test VMs or changing payload distro images.

Host routing Previous host New host Mappings
Public AzureLinux.3.Arm64.Open AzureLinux.4.Arm64.Open 8
Internal AzureLinux.3.Arm64 AzureLinux.4.Arm64 3

The 11 host-only replacements are limited to four YAML files:

File under eng/pipelines/ Preserved payload/alias coverage Mappings
coreclr/templates/helix-queues-setup.yml Alpine 3.24 ARM64, public/internal 2
libraries/helix-queues-setup.yml Ubuntu 26.04 ARM64, Azure Linux 3.0 ARM64, Alpine 3.24 ARM64 3
installer/helix-queues-setup.yml Ubuntu 26.04 ARM64 and Alpine 3.24 ARM64 2
helix-platforms.yml Alpine ARM64 latest/oldest, public/internal aliases 4

Every descriptive alias, registry/image tag/digest, distro/version/architecture identity, lane condition, queue variable/group and lane count remains unchanged. In particular, (AzureLinux.3.0.ArmArch.Open) and its azurelinux-3.0-helix-arm64v8 image remain Azure Linux 3 coverage; only their container host moves.

All 14 ARM32-on-ARM64 container mappings and both standalone CoreCLR ARM64 VM mappings retain their existing hosts. Build agents, cross-build images, SuperPMI, GitHub Actions and other platforms are unchanged. The standalone AL4 VM libmsquic prerequisite is not bypassed by this change.

Related to #136313, H1. This PR does not close the tracking issue or declare H1, S1 or S2 complete.

Validation

Static checks on commit 8869ad3d623ff1910b927f74e46adf995debac4b:

  • git diff HEAD^ HEAD --check - passed.
  • ruby -ryaml -e 'ARGV.each { |path| YAML.load_file(path); puts "PASS #{path}" }' eng/pipelines/coreclr/templates/helix-queues-setup.yml eng/pipelines/helix-platforms.yml eng/pipelines/installer/helix-queues-setup.yml eng/pipelines/libraries/helix-queues-setup.yml - passed; all four files parsed.
  • An ad hoc Ruby/Open3 full-file invariant check compared each committed file with its parent after applying only the inventoried native ARM64 host-component replacements - passed. It asserted the exact four-file set, per-file replacement counts, eight public/three internal mappings, unchanged image/alias/condition bytes, and unchanged 14 ARM32/two standalone ARM64 mappings.
  • GitHub's comparison against main reports one commit, four changed files and 11 additions/11 deletions.

Refreshed the live Helix queue catalog on 2026-10-23:

Queue Architecture IsAvailable IsInternalOnly
azurelinux.4.arm64 ARM64 true true
azurelinux.4.arm64.open ARM64 true false

Queue availability does not establish capacity or native workload/container/kernel compatibility. No product build, native Helix execution, Docker activity, manual CI trigger/retry or Helix submission was performed for this metadata-only change.

Required native CI coverage and existing gaps

Standard CI selects only part of the changed mappings; it must not be treated as complete coverage of this PR:

  • Default runtime includes CoreCLR-selected Alpine ARM64 coverage and libraries Azure Linux 3 ARM64 coverage. Confirm actual execution on the new hosts, including the applicable public/internal routes.
  • runtime-extra-platforms selects Ubuntu ARM64 libraries coverage, but its current platform matrix does not list linux_musl_arm64.
  • The libraries Alpine ARM64 branch needs includeAllPlatforms=true coverage, such as the scheduled rolling libraries outerloop path. A normal extra-platforms run alone does not exercise it.
  • Current callers of the installer queue template select no ARM64 platforms. The changed installer ARM64 mappings therefore require separately arranged native ARM64 execution; Alpine additionally requires isExtraPlatformsBuild=true or includeAllPlatforms=true.
  • No direct YAML consumers were found for the central Alpine ARM64 latest/oldest aliases. Updating those definitions is not evidence that they executed.

No coverage wiring is expanded in this PR. Native execution evidence and merge remain pending; static checks and queue availability do not complete the host migration.

Reference: OS onboarding guide.

Note

This PR description was generated with GitHub Copilot.

Update only container-host queues while preserving payload images, aliases, routing, ARM32 hosts, and standalone ARM64 VM coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 96a5c84b-34cb-4531-b8cf-e6502299ec1d
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

@steveisok
steveisok requested a review from richlander October 23, 2026 21:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human sign-off is needed on native ARM64 execution evidence, including mappings outside standard CI coverage, because static checks do not establish host/container compatibility.

No open issues 🎉

What changed in this PR

Moves native ARM64 Helix container hosts from Azure Linux 3 to Azure Linux 4, extending the earlier x64 migration without changing the distributions tested inside containers.

Changes:

  • Updates 11 host mappings: eight public and three internal.
  • Preserves payload images, digests, aliases, and lane conditions.
  • Leaves ARM32 container mappings, standalone ARM64 VMs, and build agents unchanged.
File Description
eng/​pipelines/​libraries/​helix-queues-setup.yml Moves Ubuntu, Azure Linux 3, and Alpine ARM64 container hosts to AL4.
eng/​pipelines/​installer/​helix-queues-setup.yml Moves Ubuntu and Alpine ARM64 container hosts to AL4.
eng/​pipelines/​helix-platforms.yml Updates public/internal hosts for Alpine ARM64 latest/oldest aliases.
eng/​pipelines/​coreclr/​templates/​helix-queues-setup.yml Updates public/internal hosts for Alpine ARM64 tests.

Review effort: Balanced

@steveisok
steveisok merged commit 03b9efe into dotnet:main Oct 24, 2026
187 of 189 checks passed
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Oct 26, 2026
steveisok added a commit that referenced this pull request Nov 5, 2026
<!-- -->

## Summary

Prepare the **H2 ARM32 host-only candidate** from
[#136313](#136313), following
the native ARM64 container-host changes merged in
[#136387](#136387).

Change only the host component `AzureLinux.3.Arm64[.Open]` to
`AzureLinux.4.Arm64[.Open]` for existing Debian 13 and Alpine 3.24 ARM32
containers:

| File | Host mappings |
|---|---:|
| `eng/pipelines/coreclr/templates/helix-queues-setup.yml` | 4 |
| `eng/pipelines/libraries/helix-queues-setup.yml` | 1 |
| `eng/pipelines/installer/helix-queues-setup.yml` | 1 |
| `eng/pipelines/helix-platforms.yml` | 8 |

The 14 replacements comprise eight Debian and six Alpine mappings, with
eight public and six internal queue strings. Parenthesized aliases,
public/internal routing, image tags, all six digest pins, existing
tag-only aliases, payload architectures, ABI/sysroots, cross-build
inputs, conditions, queue groups and lane counts are unchanged. All 11
native ARM64 container-host mappings from #136387, both bare AL3 ARM64
VM entries, other platforms, build agents and SuperPMI are preserved.

This is not an ARM32 image upgrade, distro-coverage change, test
conversion to ARM64, or host-tooling/provisioning change. It does not
close #136313 or declare H2, S1 or S2 complete.

## Compatibility status and merge requirement

**Actual ARM32 execution on the deployed Azure Linux 4 ARM64 Helix hosts
is unverified. This is a candidate, not a proven-compatible deployment.
Do not merge based on static checks or ARM64 queue availability alone.**

Read-only [Helix queue
metadata](https://helix.dot.net/api/2019-06-17/info/queues) advertises
gallery image `MicrosoftAzureLinux/azurelinux-4/4-arm64/4.0.2026091402`
and an ARM64 Alpine smoke image. The exposed artifact metadata does not
establish ARM32 execution or installed QEMU/binfmt support.

The [upstream AL4 kernel
configuration](https://github.com/microsoft/azurelinux/blob/1035368c970cbed17ab06a8006b6c57b25fdc856/specs/k/kernel/6.18-aarch64-azl.config#L527)
enables compat ELF support, but that is not proof of the deployed
kernel, CPU AArch32 EL0 capability, or emulation setup. Native execution
requires suitable CPU/kernel support; otherwise transparent execution
requires an already-provisioned emulator and appropriate binfmt
registration/interpreter access. No such configuration is changed here.

A bounded sample of 100 recent public ARM32 Helix jobs contained no AL4
ARM64 execution. Successful existing [Debian
ARM32](https://helix.dot.net/api/2019-06-17/jobs/b66ca86e-904b-4af4-917c-93f808a1ab67/details)
and [Alpine
ARM32](https://helix.dot.net/api/2019-06-17/jobs/7f9bae7f-55f8-42d4-9e0a-804d67b96323/details)
jobs ran on AL3, not AL4. Neither those passing jobs, kernel
`CONFIG_COMPAT`, native ARM64 success, nor
`Architecture=ARM64`/`IsAvailable=true` establishes ARM32-on-AL4
compatibility. **No concrete incompatibility has been established
either.**

Actual Debian and Alpine ARM32 execution on AL4 hosts is required before
merge. Confirm the resolved host queue, unchanged ARM32 container, and
successfully executed ARM32 workloads, rather than submission-only or
ARM64 results:

- [ ] `runtime`: `build_linux_arm_checked_CoreCLR_ReleaseLibraries` —
Debian ARM32 runtime innerloop, via the CoreCLR queue template
(`eng/pipelines/runtime.yml:663-703`).
- [ ] `runtime`: `build_linux_musl_arm_checked_CoreCLR_ReleaseLibraries`
— Alpine ARM32 libraries on Checked CoreCLR, via the CoreCLR queue
template (`eng/pipelines/runtime.yml:704-743`).
- [ ] `runtime-extra-platforms`: `linux_arm` Release
`Libraries_Release_CoreCLR` — the separately gated Debian ARM32
libraries mapping
(`eng/pipelines/extra-platforms/runtime-extra-platforms-other.yml:15-44`).
Ordinary `runtime` PR validation does not exercise this
libraries-template entry.

These are existing routes, not new lanes. No manual CI trigger/retry or
Helix submission has been performed or is requested by this change; any
manual execution requires separate authorization. Additional existing
coverage includes `runtime-coreclr outerloop` Checked JIT runs with
`platformGroup: all` for both ARM32 platforms, and scheduled libraries
outerloop Debian ARM32 runs with `includeAllPlatforms: true`.

## Coverage gaps

- **Internal queues:** public CI does not validate the internal hosts.
CoreCLR explicitly selects non-`.Open` Debian/Alpine entries; libraries
and installer senders strip `.Open` for internal projects. Those routes
are preserved, but no internal execution evidence was obtained.
`runtime-official.yml` is publication-only, not an ARM32 test route.
- **Installer:** the ARM32 mapping has no current active caller. The
three installer queue-template invocations in `runtime.yml` select other
platforms. Ordinary `runtime` or extra-platforms success cannot validate
ARM32 installer execution.
- **Central aliases:** only `helix_linux_arm32_oldest` has in-repository
consumers: cDAC dump/stress/xplat routes in `runtime-diagnostics`, plus
optional SOS ARM32 selection (excluded from default `sosPlatforms`).
These are not covered by ordinary `runtime`/extra-platforms. The
remaining seven aliases, including all internal and all Alpine ARM32
aliases, have no in-repository consumers.

No coverage expansion or host support installation is included to hide
these gaps. Closing them requires separately approved work; public CI
success must not be described as exercising every changed mapping.

## Validation

- **Passed:** `python3 "$SESSION_FILES/verify-arm32-hosts.py"` using
existing PyYAML 6.0.3. The session-local assertion script checks
duplicate-key-rejecting YAML parsing, parsed structure equality modulo
the permitted host transformation, exact full-file byte equality against
that transformation, all 14 replacements, unchanged payload
alias/image/digest bytes, preservation of all 11 landed native ARM64
mappings and both bare AL3 ARM64 entries, and the absence of unrelated
tracked changes.
- **Passed:** `git diff --check`.
- **Passed:** read-only MCR tag/manifest/config queries confirmed both
existing pinned payloads are `linux/arm/v7`; no Docker commands or image
operations were performed. Existing unpinned aliases remain floating
tags.
- **Not run:** product builds/tests (queue-metadata-only change), Azure
Pipelines template expansion, or actual ARM32 execution on AL4 Helix
hosts. No manual CI/Helix jobs, Docker operations, provisioning,
kernel/binfmt changes, or emulation installation were performed.

See the [OS onboarding
guide](https://github.com/dotnet/runtime/blob/main/docs/project/os-onboarding.md)
for the container-host testing model.

> [!NOTE]
> This pull request description and the code changes were prepared with
GitHub Copilot.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b085338b-95ae-40c2-a1e0-d77e477f5cfc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants